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(54) Method for forking or migrating a virtual machine 



(57) A method is provided for increasing the efficien- 
cy of virtual machine processing. A parent virtual ma- 
chine is provided on a host computer. The parent virtual 
machine is temporarily or permanently suspended. A 



child virtual machine is created at a new location by fork- 
ing the parent virtual machine. The child virtual machine 
may not initially include all the stored data that is asso- 
ciated with the parent virtual machine. 
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Description 

TECHNICAL FIELD OP THE INVENTION 

[0001] The present invention relates in general to the 
field of virtual machines and, more particularly, to a 
method for applying the concepts of forking and migra- 
tion to virtual machines. 

BACKGROUND OF THE INVENTION 

[0002] Computers include general purpose central 
processing units (CPUs) that are designed to execute a 
specific set of system instructions. A group of proces- 
sors that have similar architecture or design specifica- 
tions may be considered to be members of the same 
processor family. Examples of current processor fami- 
lies include the Motorola 680X0 processor family man- 
ufactured by Motorola, Inc. of Phoenix, Arizona; the Intel 
80X86 processor family, manufactured by Intel Corpo- 
ration of Sunnyvale, California; and the PowerPC proc- 
essor family, which is manufactured by Motorola, Inc 
and used in computers manufactured by Apple Compu- 
ter, Inc. of Cupertino, California. Although a group of 
processors may be in the same family because of their 
similar architecture and design considerations, proces- 
sors may vary widely within a family according to their 
clock speed and other performance parameters. 
[0003] Each family of microprocessors executes in- 
structions that are unique to the processor family. The 
collective set of instructions that a processor or family 
of processors can execute is known as the processor's 
instruction set. As an example, the instruction set used 
by the Intel 80X86 processor family is incompatible with 
the instruction set used by the PowerPC processorfam- 
ily. The Intel 80X86 instruction set is based on the Com- 
plex Instruction Set Computer (CISC) format. The Mo- 
torola PowerPC instruction set is based on the Reduced 
Instruction Set Computer (RISC) format. CISC proces- 
sors use a large number of instructions, some of which 
can perform rather complicated functions, but which re- 
quire generally many clock cycles to execute. RISC 
processors use a smaller number of available instruc- 
tions to perform a simpler set of functions that are exe- 
cuted at a higher rate. 

[0004] The uniqueness of the processor family among 
computer systems also typically results in incompatibil- 
ity among the other elements of hardware architecture 
of the computer systems. A computer system manufac- 
tured with a processor from the Intel 80X86 processor 
family will have a hardware architecture that is different 
from the hardware architecture of a computer system 
manufactured with a processor from the PowerPC proc- 
essor family. Because of the uniqueness of the proces- 
sor instruction set and a computer system's hardware 
architecture, application software programs are typically 
written to run on a particular computer system running 
a particular operating system. 
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[0005] A computer manufacturer will want to maxi- 
mize its functionality by having more rather than fewer 
applications run on the microprocessor family associat- 
ed with the computer manufacturer's product line. To ex- 
5 pand the number of operating systems and application 
programs that can run on a computer system, a field of 
technology has developed in which a given computer 
having one type of, CPU, called a host, will run an em- 
ulator program that allows the host computer to emulate 
io receiving and executing the instructions of an unrelated 
type of CPU, called a guest. Thus, the host computer 
will execute an application that will cause one or more 
host instructions to be called in response to a' given 
guest instruction. In some cases, the host computer can 
*5 both run software designed for its own hardware archi- 
tecture, other than the emulation program, and software 
written for computers having an unrelated hardware ar- 
chitecture. As a more specific example, a computer sys- 
tem manufactured by Apple Computer, for example 
may run operating systems and programs written for 
PC-based computer systems. It may also be possible to 
use an emulator program to concurrently operate mul- 
tiple incompatible operating systems on a single CPU 
In this arrangement, although each operating system is 
25 incompatible with the other, an emulator program can 
host one of the two operating systems, allowing the oth- 
erwise incompatible operating systems to run concur- 
rently on the same computer system. 
[0006] When a guest computer system is emulated on 
30 a host computer system, the guest computer system is 
said to be a virtual machine, as the guest computer sys- 
tem exists only as a software representation of the op- 
eration of the hardware architecture of the guest com- 
puter system. The terms emulator and virtual machine 
35 are sometimes used interchangeably to denote the abil- 
ity to mimic or emulate the hardware architecture of an 
entire computer system. As an example, the Virtual PC 
software created by Connectix Corporation of San 
Mateo, California emulates an entire computer that in- 
40 eludes an Intel 80X86 Pentium processor and various 
motherboard components and cards. The operation of 
these components is emulated in the virtual machine 
that is being run on the host machine. An emulator pro- 
gram executing on the operating system software and 
45 hardware architecture of the host computer such as a 
computer system having a PowerPC processor, mimics 
the operation of the entire guest computer system The 
emulator program acts as the interchange between the 
hardware architecture of the host machine and the in- 
50 structions transmitted by the software running within the 
emulated environment. 

[0007] One advantage of a virtual machine over a real 
machine is the ability to quickly and cheaply create mul- 
tiple instances of virtual machines. If allowed by the vir- 
55 tual machine implementation, multiple virtual machines 
can exist simultaneously in a single host machine (host 
computer system) environment. Resources of the host 
machine can be divided among the various virtual ma- 



2 



BNSDOCID: <EP 1380947A2 ..I. > 



3 



EP 1 380 947 A2 



4 



chines. For example, a single host machine with four 
processors and 1 gigabyte of random access memory 
(RAM) could be divided evenly into four virtual ma- 
chines; each of which is given one processor and 256 
megabytes of RAM. Other resource allocation divisions 
are possible. 

[0008] This flexible resource allocation becomes 
even more useful when combined with the ability to 
move virtual machines from one host machine to anoth- 
er. This allows for "load balancing" of systems. For ex- 
ample, if a virtual machine requires more processing 
power than is available on one host machine, it can be 
moved to another host machine that has extra capacity. 
[0009] In some computing environments, it is useful 
to have multiple machines that are nearly identical in 
configuration (both hardware and software). For exam- 
ple, a large electronic-commerce web site such as Am- 
azon.com has dozens or hundreds of web servers that 
are all nearly identical in configuration. This setup allows 
for easy expandability. When current capacity is inade- 
quate, additional servers can quickly be brought on line. 
[0010] Another case where nearly-identical configu- 
rations are useful is in the testing of configuration mod- 
ifications. When dealing with mission-critical applica- 
tions, IS managers often want to test software configu- 
ration changes before applying them to the production 
system. For example, if a new "security patch" was 
made available by Microsoft Corporation for the Win- 
dows operating system, an administrator may want to 
test this patch on a separate server machine before in- 
stalling the patch on the production server. 

SUMMARY OF THE INVENTION 

[0011] The present invention in one implementation 
provides a method for increasing the efficiency of virtual 
machine processing. One step of the method is provid- 
ing a parent virtual machine. Another step is temporarily 
suspending the parent virtual machine. Another step is 
forking the parent virtual machine to create a child virtual 
machine at a new location. 

[0012] The present invention provides another meth- 
od for increasing the efficiency of virtual machine 
processing. One step of the method is providing a parent 
virtual machine that is associated with stored data. An- 
other step of the method is temporarily suspending the 
parent virtual machine. Another step is forking the par- 
ent virtual machine to create a child virtual machine at 
a new location without at least a first portion of the stored 
data. 

[0013] Further, the present invention provides anoth- 
er method for increasing the efficiency of virtual machine 
processing. One step of the method is providing a parent 
virtual machine that is associated with stored data. An- 
other step of the method is permanently suspending the 
parent virtual machine. Another step is migrating the 
parent virtual machine to create a child virtual machine 
at a new location without at least a first portion of the 



stored data. 

[0014] An advantage of one implementation of the 
present invention is that it makes possible the forking of 
a virtual machine. The ability to fork increases the utility 

5 of the aforementioned multiple instances of virtual ma- 
chines. Forking can, in some circumstances, create 
multiple instances quickly enough that most functions of 
and applications running on the virtual machines are not 
affected significantly. 

10 [0015] Another advantage of an implementation of the 
present invention is that it increases the efficiency of vir- 
tual machine processing. 

[0016] One implementation of the method of the 
present invention has the advantage of a forking time 
'5 unaffected by the size of the parent virtual machine's 
memory. When the non-memory portion of the parent 
virtual machine consists of only a few hundred kilobytes 
of data, the forking time is about a millisecond on some 
systems. 

20 [001 7] If demand paging is added to the method, then 
total execution time may also depend on the size of the 
parent's memory. However, demand paging does not 
cause conflicts with the applications running on the par- 
ent and child virtual machines. 

25 [001 8] No one of the preceding advantages is critical 
to the invention. Particular implementations of the inven- 
tion may achieve only a subset of the advantages. For 
example, one implementation of the invention may only 
provide the option of forking a virtual machine. Other 

30 technical advantages of the present invention will be 
readily apparent to one skilled in the art from the follow- 
ing figures, descriptions, and claims. 

DETAILED DESCRIPTION OF THE INVENTION 

35 

[0019] In the case of an emulated computer system 
or virtual machine, an emulation program provides an 
emulated operating environment in the host computer 
system. Shown in Figure 1 is a diagram of the logical 

^o layers of the hardware and software architecture for an 
emulated operating environment in a computer system 
10. An emulation program 14 runs on a host operating 
system that executes on the host computer system 
hardware or processor 11 . Emulation program 14 emu- 

45 lates a guest computer system 1 6, including a guest op- 
erating system 18. Guest application programs are able 
to execute on guest operating system 18. In the emu- 
lated operating environment of Figure 1 , because of the 
operation of emulation program 14, guest application 20 

so can run on the computer system 1 0 even though guest 
application 20 is designed to run on an operating system 
that is generally incompatible with host operating sys- 
tem 12 and host computer system hardware 11 . 
[0020] Referring now to Figures 2 and 3, "forking" is 

55 a term used by UNIX programmers to describe the du- 
plication of a UNIX process and its address space. Both 
the original process and the fork are then allowed to run 
as independent processes from the forking point. The 
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implementation of forking often involves a technique 
called "copy on write" in which case all memory pages 
in both address spaces are marked "write protected". 
When either the original or the forked process writes to 
a page, a copy is made so that each process has its own 
copy. Pages thai are not modified can continue to be 
shared between the two processes. This technique not 
only saves on memory resources, but It also makes fork- 
ing much faster than otherwise possible. 
[0021] In the present invention, the concept of forking 
a single process is combined with the concept of a virtual 
machine. The present invention allows for fast conver- 
sion of shared resources to private copies between an 
original virtual machine and its fork. However, resource 
sharing is possible only if both virtual machines are run- 
ning on the same host. 

[0022] Virtual machine forking can be used to quickly 
replicate an existing virtual machine. For example, if a 
user wants to test a new patch, he can fork the virtual 
machine and apply the patch to the non-production fork. 
Once the patch has been tested, it can be applied to the 
original copy with limited risk. Alternatively, the produc- 
tion and non-production virtual machines could be 
swapped once the testing of the patch was completed. 
[0023] Shown in Figure 2 is a flow diagram of a meth- 
od 20 for forking a virtual machine. In step 202, a virtual 
machine parent is suspended. In step 204, a copy or 
"snapshot" is made of all of the pieces of the parent vir- 
tual machine other than the memory of the parent virtual 
machine. In step 206, the snapshot is moved to a new 
location, i.e. a location other than the location of the par- 
ent. Moving the snapshot to a new location creates a 
new virtual machine child. The child may or may not be 
located on the same host computer system as the par- 
ent. In step 208, pieces of the parent's memory are sent 
to the child using demand paging. Demand paging, de- 
picted in Figure 3, is a method for sending pieces or pag- 
es of memory from the parent to the child. In demand 
paging, parent memory is prioritized based on what the 
child actively requires. 

[0024] Figure 3 is a flow diagram of the demand pag- 
ing of step 208 used in forking a virtual machine. In step 
302, it is determined whether or not the parent virtual 
machine is about to modify a piece of the parent's mem- 
ory that has not yet been sent to the child. If the parent 
virtual machine is about to modify a memory piece, in 
step 304, then before the parent is allowed to modify the 
memory piece, either the piece is sent to the child virtual 
machine or a temporary copy of the piece is made. The 
temporary copy is saved by the parent until a later, as- 
yet-undetermined point in the demand paging method, 
at which point the copy is sent to the child. If the parent 
is not about to modify a memory piece or if step 304 is 
completed, the method proceeds to step 306. 
[0025] In step 306, it is determined whether or not the 
child virtual machine is accessing the memory of the 
parent virtual machine. If the child is accessing the par- 
ent's memory, in step 308, the child virtual machine is 



temporarily suspended and the piece of the parent's 
memory required by the child is sent from the parent to 
the child. If the child is not accessing the parents mem- 
ory, in step 310, pieces of the parent's memory that are 
5 not actively required by the child may be sent from the 
parent to the child. If step 308 or step 310 is completed, 
the method proceeds to step 312. 
[0026] In step 312, it is determined whether or not all 
of the memory of the parent virtual machine has been 
io sent to the child virtual machine. If not all of the memory 
has been sent, the method proceeds to step 302, de- 
scribed above. If all of the memory has been sent, then 
the demand paging method ends. 
[0027] Referring now to Figures 4 and 5, "migrating" 
15 is a term that refers to the movement of a virtual machine 
from one host machine to another. When a virtual ma- 
chine is migrated, the original virtual machine is perma- 
nently suspended: and the copy runs in a new location. 
One way to implement migration involves saving the en- 
20 tire state of the virtual machine (including all of its RAM) 
to a file on disk, then copying the file to the new host 
and restoring the machine state. 
[0028] In the present invention, migrating is similar to 
forking in its implementation. As with a forked virtual ma- 
25 chine, a migrated virtual machine can start running al- 
most immediately after it is created, i.e. within a second 
or two after creation. This feature results from the use 
of demand paging and "copy-on-access," wherein copy- 
on-access is defined as the copying of parent's memory 
30 to the child upon the child's accessing the parent's mem- 
ory. Copy-on-access is an aspect of demand paging for 
migrating a virtual machine. 

[0029] Copy-on-access is also an advantage of the 
present invention over existing technologies. Existing 

35 technologies save and restore an entire copy of the par- 
ent's Random Access Memory (RAM). Existing technol- 
ogies' saving and restoring can take anywhere from 5 
to 60 seconds, depending on the size of the memory 
allocated to the parent 

^0 [0030] One application of migrating a virtual machine 
is load balancing. Another application would be for fail- 
over or hardware maintenance. For example, if the hard- 
ware in the host machine requires maintenance (e.g. 
more memory is to be added), the virtual machine can 

45 be temporarily migrated to a backup machine, prevent- 
ing downtime. 

[0031] Shown in Figure 4 is a flow diagram of a meth- 
od 40 for migrating a virtual machine. In step 402, a par- 
ent virtual machine is permanently suspended. In step 

50 204, a copy or "snapshot" is made of all of the pieces of 
the parent virtual machine other than the memory of the 
parent virtual machine. In step 206, the snapshot is 
moved to a new location, i.e. a location other than the 
location of the parent. Moving the snapshot to a new 

55 location creates a child virtual machine. The child may 
or may not be located on the same computer system as 
the parent. In step 404, pieces of the parent's memory 
are sent to the child using demand paging. Demand 
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chine processing, comprising the steps of: 

providing on a first host system a parent virtual 
machine; 

5 temporarily suspending the parent virtual ma- 

chine; 

forking the parent virtual machine to create at 
least one child virtual machine at an at least one 
new location. 

10 

2. The method of claim 1 , wherein the at least one new 
location is on the first host computer system. 

3. The method of claim 1 , wherein the at least one new 
1 * location is on a second host computer system. 

4. The method of claim 1 , wherein the steps of tempo- 
rarily suspending the parent virtual machine and 
forking the parent virtual machine are executed in 

20 a time duration of about one millisecond. 

5. The method of claim 1 , wherein the parent virtual 
machine is a network server. 

25 6. A method for increasing the efficiency of virtual ma- 
chine processing, comprising the steps of: 

providing on a first host system a parent virtual 
machine associated with stored data; 
30 temporarily suspending the parent virtual ma- 

chine; 

forking the parent virtual machine to create at 
least one child virtual machine at an at least one 
new location without a first portion of the stored 
35 data. 

7. The method of claim 6, wherein the at least one new 
location is on the first host computer system. 

40 8. The method of claim 6, wherein the at least one new 
location is on a second host computer system. 
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paging, depicted in Figure 5, is a method for sending 
pieces or pages of memory from the parent to the child. 
In demand paging, parent memory is prioritized based 
on what the child actively requires. 
[0032] Figure 5 is a flow diagram of the demand pag- 
ing of step 404 used in migrating a virtual machine. In 
step 306, it is determined whether or not the child virtual 
machine is accessing the memory of the parent virtual 
machine. If the child is accessing the parent's memory, 
in step 308, the child virtual machine is temporarily sus- 
pended and the piece of the parent's memory required 
by the child is sent from the parent to the child. If the 
child is not accessing the parent's memory, in step 310, 
pieces of the parent's memory that are not actively re- 
quired by the child may be sent from the parent to the 
child. If step 308 or step 310 is completed, the method 
proceeds to step 31 2. 

[0033] In step 31 2, it is determined whether or not all 
of the memory of the parent virtual machine has been 
sent to the child virtual machine. If not all of the memory 
has been sent, the method proceeds to step 306, de- 
scribed above. If all of the memory has been sent, the 
method proceeds to step 502. In step 502, the parent's 
memory is deleted. After step 502 ; the demand paging 
method ends. 

[0034] The present invention is not limited in its appli- 
cation to the emulation of a particular computer system 
architecture, particularly the Intel 80X86 architecture. 
[0035] Although the present invention has been de- 
scribed in detail, it should be understood that various 
changes, substitutions, and alterations can be made 
thereto without departing from the spirit and scope of 
the invention as defined by the appended claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0036] A more complete understanding of the present 
invention and advantages thereof may be acquired by 
referring to the following description taken in conjunction 
with the accompanying drawings, in which like reference 
numbers indicate like features, and wherein: 

Figure 1 is a diagram of the logical relationship of 
the elements of an emulated computer system run- 
ning in a host computer system; 
Figure 2 is a flow diagram of a method for forking a 
virtual machine; 

Figure 3 is a flow diagram of demand paging used 
in forking a virtual machine; 
Figure 4 is a flow diagram of a method for migrating 
a virtual machine; and 

Figure 5 is a flow diagram of demand paging used 
in migrating a virtual machine. 



Claims 

1 . A method for increasing the efficiency of virtual ma- 



9. The method of claim 6, wherein the steps of tempo- 
rarily suspending the parent virtual machine and 
making the copy of the parent virtual machine are 
executed in a time duration of about one millisec- 
ond. 

10. The method of claim 6, wherein the parent virtual 
machine is a network server. 

11. The method of claim 6, further comprising the step 
of: 

using demand paging to send subportions of 
the first portion of the stored data to the at least 
one child virtual machine. 
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12. The method of claim 11 , wherein the step of using 
demand paging comprises the step of: 

if the at least one child virtual machine is ac- 
cessing a first sub-portion of the first portion, 
temporarily suspending the at least one child 
virtual machine and sending the first subporlion 
to the at least one child virtual machine. 

13. The method of claim 12, wherein the step of using 
demand paging further comprises the step of. 

if not all of the first portion of the stored data 
associated with the parent virtual machine has 
been sent to the at least one child virtual ma- 
chine, sending a second subportion of the first 
portion to the at least one child virtual machine. 

14. The method of claim 11 , wherein the step of using 
demand paging comprises the step of: 

if the at least one child virtual machine is not 
accessing the at least one memory of the par- 
ent virtual machine, sending memory pages of 
the first portion to the at least one child virtual 
machine. 

15. The method of claim 6, wherein the first portion is 
all the stored data associated with the parent virtual 
machine. 

16. The method of claim 11 , wherein the step of using 
demand paging comprises the step of: 

if the parent virtual machine attempts to modify 
a first subportion, sending the first subportion 
to the at least one child virtual machine before 
the first subportion is modified by the parent vir- 
tual machine. 

17. The method of claim 16, wherein the step of using 
demand paging further comprises the step of: 

if not all of the first portion of the stored data 
associated with the parent virtual machine has 
been sent to the at least one child virtual ma- 
chine, sending a second subportion of the first 
portion to the at least one child virtual machine. 

18. The method of claim 11 , wherein the step of using 
demand paging comprises the step of: 
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20 



25 



first subportion is a temporary copy. 

20. The method of claim 18, wherein the step of using 
demand paging further comprises the step of: 

if not all or the first portion of the stored data 
associated with the parent virtual machine has 
been sent to the at least one child virtual ma- 
chine, sending a second subportion of the first 
portion to the at least one child virtual machine. 

21 . A method for increasing the efficiency of virtual ma- 
chine processing, comprising the steps of: 

providing on a first host system a parent virtual 
machine associated with stored data; 
permanently suspending the parent virtual ma- 
chine; 

migrating the parent virtual machine to create 
at least one child virtual machine without a first 
portion of the stored data at an at least one new 
location. 

22. The method of claim 21 , wherein the at least one 
new location is on the first host computer system. 



23. 



30 24. 



The method of claim 21, wherein the at least one 
new location is on a second host computer system. 

The method of claim 21 , wherein the steps of per- 
manently suspending the parent virtual machine 
and migrating the parent virtual machine are exe- 
cuted in a time duration of about one millisecond. 



35 25. The method of claim 21 , wherein the parent virtual 
machine is a network server. 

26. The method of claim 21 , further comprising the step 
of: 

40 

deleting the parent virtual machine. 

27. The method of claim 21 , further comprising the step 
of: 



45 



50 



using demand paging to send the first portion 
of the stored data to the at least one child virtual 
machine. 



if the parent virtual machine attempts to modify 
a first subportion, making a copy of the first sub- 
portion before the first subportion is modified 55 
by the parent virtual machine. 

19. The method of claim 18, wherein the copy of the 
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